Images captured by camera or other video and image systems usually contain noise introduced by analog devices or during conversion from analog signal to digital signal. Noise can be a particularly significant problem in low light conditions. Thus image denoising technologies have been developed to improve signal quality of captured images. Depending on the data being used, image denoising methods can be divided into spatial denoising methods which use intra-frame information only, temporal denoising methods that employ inter-frame information and motion compensation, and methods that combine both intra-frame and inter-frame information for denoising.
BM3D (block-matching three-dimensional) method is developed to perform image denoising. By block-matching (BM), similar 2D (two-dimensional) patches of each 2D reference patch are identified for denoising purpose. The similar 2D patches are selected from the still image or the video frame containing the reference patch by spatial denoising methods. For temporal denoising methods, the similar patches can also be found from a sequence of digital video frames. The reference patch and the matched patches can be grouped as a patch set according to the similarity of the grouped patches. The patch set is then transformed and denoised. The weighted average of the reference patch and its similar patches (which are also called matched patches) can be used to reduce noise of the all pixels of the reference patch.
FIG. 1 illustrates an example of image analysis by BM to identify similar patches and group the patches into patch sets, each of which can be represented as a 3D data array. As shown by FIG. 1, one noisy image is processed by BM and each processed patch and the similar patches are grouped into 3D arrays. Each processed patch by BM is also treated as a reference patch. Reference patch R10 and the matched patches M11 and M12 can be grouped to form 3D data array 110. Reference patch R20 has four matched patches M21, M22, M23 and M24 with similar image content. These five patches are stacked into 3D data array 120. Patch R30 and the matched patches M31, M32 and M33 are grouped into 3D data array 130. Patch R40 together with its matched patch M41, M42 and M43 are grouped into 3D data array 140. In each 3D data array, the correlation between the patches of each 3D array can be employed to improve performance of image processing.
Based on the denoised result using block-matching, the noise of an image is attenuated in transform domain. FIG. 2 illustrates an exemplary flow chart of image denoising by a BM3D method. The denoising on noisy image 210 is performed by two denoising processes. The first is denoising process 220 which performs image denoising by hard-thresholding and the second is process 230 which performs image denoising by empirical Wiener filtering. In process 220, grouping by block-matching is performed on noisy image 210 to search for similar patches of the current patch by process 221. The current patch of original noisy image is used as a reference patch referred to as R1. Reference patch R1 together with its similar patches is grouped into one 3D data array by process 222. Then the 3D data array is processed by transform-domain shrinkage which starts from 3D transform 223. The 3D transform corresponds to a unitary transform which can be implemented as 2D discrete cosine transform (DCT) followed by 1D Hadamard transform. The similarity or correlation between the patches in the 3D array is exploited by applying the 3D unitary transform to produce a sparse representation of the true signal in 3D transform domain. Then the noise is attenuated by hard-thresholding 224 on the transform coefficients in 3D domain and the corresponding weight is obtained. Inverse 3D transform 225 following hard-thresholding 224 is performed to produce estimates of all the patches. All patches of the noisy image are processed in a sliding manner to yield block-wise estimates of all the patches. The overlapping block-wise estimates for each reference block forms a redundant estimate of the true signal. In order to get a non-redundant estimate of the noisy image, aggregation 226 is performed on the block-wise estimates and the weight produced by hard-thresholding 224. The basic estimate of the noisy image is obtained by aggregation 226 in denoising step 220.
The basic estimate (i.e. weighted average of all patches) of the noisy image is used by the second denoising process. Instead of using hard-thresholding, Wiener filtering is used to reduce noise in denoising process 230. The basic estimate of R1 is identified as reference patch R2. Grouping by block-matching 231 is performed within the basic estimate rather than within noisy image 210 to find similar patches of reference patch R2. R2 and the similar patches are stacked together to form one 3D data array by processing 232. The 3D data is processed by 3D transform 233 including 2D DCT and 1D Hadamard transform. The transform coefficients are processed by Wiener filtering 234 to attenuate the noise in transform domain and produce weight of each patch. Then 3D inverse transform 235 is performed to yield estimates for all patches. All the patches of the image are processed in a sliding manner to generate block-wise estimates. The final Wiener estimate is computed by aggregation 236 which is based on the weight obtained from Wiener filtering 234 and the block-wise estimates output by inverse 3D transform 235.
During image denoising in BM3D processing by hard-thresholding such as process 220 shown in FIG. 2, noise variance is utilized on the shrinkage of transform coefficients to realize denoising.
FIG. 3 illustrates an exemplary flow of denoising by hard-thresholding in 3D transform domain with block-matching. Noise variance is estimated based on input noisy image. As shown by FIG. 3, noisy image 310 is processed first to estimate variance σ2 by variance estimation 311. Based on the estimation result, the standard deviation σ is computed in process 312. The standard deviation σ can be output to block-matching process 320 and denoising process 330 which shrinks transform coefficients by hard-thresholding. Block-matching is performed on noisy image 310 to yield the block-matching result SχR. The block-matching result SχR contains the coordinates of the matched patches that are similar to the reference patch referred as R. The block-matching result and the standard deviation σ are used by denoising process 330 to perform denoising by hard-thresholding in 3D transform domain.
Based on the result of block-matching, each reference patch and the matched patches are grouped to form 3D data array ZSχR. The denoising process by hard-thresholding in 3D transform domain is performed on 3D array ZSχR. The denoising process in 3D transform domain comprises three steps which are 3D transform, hard-thresholding and inverse 3D transform. The 3D transform produces a spares representation of the true signal in transform domain based on the similarity among the grouped patches. By hard-thresholding the transform coefficients, corresponding weight of each grouped patches is produced and the noise of grouped patches is attenuated according to the value of standard deviation σ. Inverse 3D transform follows hard-thresholding to reconstruct estimate of each patch. Denoising in 3D transform domain 332 outputs weight ψSχR and 3D array ŶSχR of reconstructed patch estimates. Kaiser window Wwin2D is multiplied to weight ωSχR to yield one patch of weights W (χR) by multiplying process 341. The summations in the denominator, which are represented by wbuff; are accumulated to the weights buffer by process 343. By using Kaiser Window Wwin2D, the impact of artifacts on the borders of patches (border effects) can be reduced. Another multiplying process 342 is used to multiply the patch of weights W (χR) to the estimate of each reconstructed patch. The results of multiplying process 342 are accumulated to estimate buffer by process 344. The summations in the numerator are represented by “ebuff”. Process 345 is used to compute the intermediate estimate or basic estimate of the noisy image by dividing ebuff by wbuff. The intermediate estimate is used as basic estimate for denoising process by Wiener filtering.
In hard-thresholding of traditional BM3D methods, several related arts estimate the image noise variance of the input noisy image for denoising and other related arts assume the noise variance of input image is known. The noise variance may be defined on the input image or on a transform domain. However, all these denoising methods in traditional methods share the same shortcoming that the same noise variance is applied to the whole image regardless of how the noise variance is determined. For example, the same noise variance is used to reduce noise on all patches of the input image. Another example is to transfer the input image into a new image domain such as frequency domain. Then, the BM3D method is performed to the new image domain and utilizes the same noise variance to process all the patches of the input image. The same variance is used in the denoising process for the whole image in either temporal domain or a transform domain.
In reality, noise level varies even in the same input image due to the fluctuation of signal strength. Using the same noise variance level for denoising the whole image will apparently have problems when the actual noise variance in a patch deviates from the given noise variance. The result of denoising based on same noise variance that is far off the actual value will cause the denoised area to become more blurred. This problem motivates the current invention.